CentOS 7
Sponsored Link

KVM : GPU Passthrough
2018/01/31
 
Configure GPU Passthrough for Virtual Machines.
By this configuration, it's possible to use GPU on Virtual Machines and run GPU Computing by CUDA, Machine learning/Deep Learning by TensorFlow.
Before configuration, Enable VT-d (Intel) or AMD IOMMU (AMD) on BIOS Setting first.
[1] Enable IOMMU on KVM Host.
[root@dlp ~]#
vi /etc/default/grub
# line 6: add (if AMD CPU, add [amd_iommu=on])

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet
intel_iommu=on
"
GRUB_DISABLE_RECOVERY="true"
[root@dlp ~]#
grub2-mkconfig -o /boot/grub2/grub.cfg

[root@dlp ~]#
reboot
# after rebooting, verify IOMMU is enabled like follows

[root@dlp ~]# 
dmesg | grep -E "DMAR|IOMMU" 


[    0.000000] ACPI: DMAR 00000000bf79e0d0 00118 (v01    AMI  OEMDMAR 00000001 MSFT 00000097)
[    0.000000] DMAR: IOMMU enabled
[    0.069846] DMAR: Host address width 40
[    0.069848] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[    0.069856] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[    0.069858] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff
[    0.069859] DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
[    0.069861] DMAR: ATSR flags: 0x0
[    0.069863] DMAR-IR: IOAPIC id 6 under DRHD base  0xfbffe000 IOMMU 0
[    0.070128] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.960840] DMAR: dmar0: Using Queued invalidation
[    0.960860] DMAR: Setting RMRR:
[    0.960890] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf7ec000 - 0xbf7fffff]
[    0.960925] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf7ec000 - 0xbf7fffff]
[    0.960958] DMAR: Setting identity map for device 0000:00:1a.2 [0xbf7ec000 - 0xbf7fffff]
[    0.960994] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf7ec000 - 0xbf7fffff]
[    0.961031] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf7ec000 - 0xbf7fffff]
[    0.961064] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf7ec000 - 0xbf7fffff]
[    0.961100] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf7ec000 - 0xbf7fffff]
[    0.961132] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf7ec000 - 0xbf7fffff]
[    0.961153] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 - 0xeffff]
[    0.961167] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 - 0xeffff]
[    0.961182] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 - 0xeffff]
[    0.961196] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 - 0xeffff]
[    0.961210] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeffff]
[    0.961224] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 - 0xeffff]
[    0.961238] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 - 0xeffff]
[    0.961255] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 - 0xeffff]
[    0.961270] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.961287] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.961424] DMAR: Intel(R) Virtualization Technology for Directed I/O
[2] Enable vfio-pci kernel module. It's OK to keep common kernel provided by RHEL/CentOS official.
# OK with common kernel

[root@dlp ~]#
uname -a

Linux dlp.srv.world 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25 20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# show PCI identification number and [vendor-ID:device-ID] of Graphic card

# PCI number ⇒ it matchs [03:00.*] below, vendor-ID:device-ID ⇒ it matchs [10de:***] below

[root@dlp ~]#
lspci -nn | grep -i nvidia

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
03:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
[root@dlp ~]#
vi /etc/modprobe.d/vfio.conf
# create new: for [ids=***], specify [vendor-ID:device-ID]

options vfio-pci ids=10de:1c03,10de:10f1
[root@dlp ~]#
echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

[root@dlp ~]#
reboot
# it's OK if it's enabled like follows

[root@dlp ~]#
dmesg | grep -i vfio 

[    4.338817] VFIO - User Level meta-driver version: 0.3
[    4.348806] vfio_pci: add [10de:1c03[ffff:ffff]] class 0x000000/00000000
[    4.348958] vfio_pci: add [10de:10f1[ffff:ffff]] class 0x000000/00000000
[3] Update QEMU.
[root@dlp ~]#
/usr/libexec/qemu-kvm -version

QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-141.el7_4.6), Copyright (c) 2003-2008 Fabrice Bellard
[root@dlp ~]#
yum -y install centos-release-qemu-ev
# disable usually

[root@dlp ~]#
sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo
# for this installing, [qemu-kvm] package is replaced to [qemu-kvm-ev] package

[root@dlp ~]#
yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev
[root@dlp ~]#
systemctl restart libvirtd
[root@dlp ~]#
/usr/libexec/qemu-kvm -version

QEMU emulator version 2.9.0(qemu-kvm-ev-2.9.0-16.el7_4.13.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
# verify q35 is included like follows

[root@dlp ~]#
/usr/libexec/qemu-kvm -machine help 

Supported machines are:
pc                   RHEL 7.4.0 PC (i440FX + PIIX, 1996) (alias of pc-i440fx-rhel7.4.0)
pc-i440fx-rhel7.4.0  RHEL 7.4.0 PC (i440FX + PIIX, 1996) (default)
pc-i440fx-rhel7.3.0  RHEL 7.3.0 PC (i440FX + PIIX, 1996)
pc-i440fx-rhel7.2.0  RHEL 7.2.0 PC (i440FX + PIIX, 1996)
pc-i440fx-rhel7.1.0  RHEL 7.1.0 PC (i440FX + PIIX, 1996)
pc-i440fx-rhel7.0.0  RHEL 7.0.0 PC (i440FX + PIIX, 1996)
rhel6.6.0            RHEL 6.6.0 PC
rhel6.5.0            RHEL 6.5.0 PC
rhel6.4.0            RHEL 6.4.0 PC
rhel6.3.0            RHEL 6.3.0 PC
rhel6.2.0            RHEL 6.2.0 PC
rhel6.1.0            RHEL 6.1.0 PC
rhel6.0.0            RHEL 6.0.0 PC
q35                  RHEL-7.4.0 PC (Q35 + ICH9, 2009) (alias of pc-q35-rhel7.4.0)
pc-q35-rhel7.4.0     RHEL-7.4.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel7.3.0     RHEL-7.3.0 PC (Q35 + ICH9, 2009)
none                 empty machine
[4] It's OK all. Create a new VM with GPU.
For [--host-device], specify GPU, and for [--machine], specify [q35], for [--features], specify [kvm_hidden=on].
[root@dlp ~]#
virt-install \
--name centos7 \
--ram 8192 \
--disk path=/var/kvm/images/centos7.img,size=30 \
--vcpus 4 \
--os-type linux \
--os-variant rhel7 \
--network bridge=br0 \
--graphics none \
--console pty,target_type=serial \
--location 'http://ftp.iij.ad.jp/pub/linux/centos/7/os/x86_64/' \
--extra-args 'console=ttyS0,115200n8 serial' \
--host-device 03:00.0 \
--features kvm_hidden=on \
--machine q35
[5] After finishing create VM, verify Graphic card is shown on VM.
If no problem, try to Install Graphic Driver, GPU Computing by CUDA, Machine learning/Deep Learning by TensorFlow.
[root@localhost ~]#
lspci | grep -i nvidia

04:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
 
Tweet